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Abstract: In spring 2004, more than 1,200 teachers and principals in 
Tennessee participated in online professional development as part of the 
state’s Reading First grant. The opportunity also allowed the state to 
investigate the possibility of using online professional development to 
help teachers meet high-quality teacher requirements. Staff at the 
Appalachia Educational Laboratory (AEL) relied on internally developed 
instructional design standards and John Keller’s ARCS model of 
motivational design to develop and deliver the online professional 
development. An evaluation was completed by The CNA Corporation. 
Gains shown from pre- to posttest scores indicate that the training allowed 
participants to make significant gains in content knowledge. Three sources 
of participant feedback indicate high levels of satisfaction with the content 
of the training modules. 



Introduction 

The reauthorization of the Elementary and Secondary Education Act, or the No Child 
Left Behind Act of 2001 (NCLB), creates opportunities and challenges for schools. One 
opportunity is increased funding for state-designed literacy grants through Reading First. With 
this funding, states can design and offer reading programs to improve the skills of K-3 principals 
and teachers, and K-12 teachers of special education. Concomitant with this opportunity is the 
challenge that all schools have highly qualified teachers by 2006. In order to meet this challenge, 
teachers can turn to college coursework or professional development. States with a significant 
number of rural school districts, such as Tennessee, must find nontraditional solutions to 
providing high-quality, research-based professional development for their educators. One such 
solution is online professional development. 

Online professional development is a subset of distance learning in general, and as such 
has seen tremendous growth in terms of technologies, pedagogy, and research over the last 10 
years. In the last decade, numerous colleges and universities have undertaken efforts to make all 
of the courses in their catalogues (both graduate and undergraduate) available online, and new 
virtual universities such as Phoenix University have become major degree-granting institutions. 
Virtual high schools have been established — either for profit or as extensions of public school 
offerings — in more than a dozen states. Indeed, the online professional development market has 
become a lucrative and competitive one. Yet, even as the market experiences this growth, the 
literature indicates little research that uses the “gold standard” of randomized control studies to 
determine the effectiveness of online professional development. Some measures used to 
evaluate distance learning in general include student outcomes as well as faculty and student 
perceptions (Ramage, 2002). 

The concept of distance learning, or e-learning , as it is sometimes called, has received 
strong support from the U.S. Department of Education as one strategy for achieving several goals 
of NCLB. E-leaming can provide greater access to high-quality professional development for 
teachers, paraprofessionals, and principals, as required specifically by Title II of the Act and 
many subsections throughout the law (e.g., Reading First, Enhancing Education through 
Technology). A July 12, 2002, press release entitled “Paige Touts Technology, E-Learning on 




His No Child Left Behind Tour Across America” (U.S. Department of Education, 2002) and a 
2004 white paper (Kleiman, 2004) distributed at the Secretary’s No Child Left Behind 
Leadership Summit are but two sources indicating that e-learning is supported by the U.S. 
Department of Education. 



Background 

During 2003, the Appalachia Educational Laboratory (AEL) developed a series of five 
online professional development modules for the state of Tennessee as part of the state’s Reading 
First grant. The five modules comprised a single course, Assessment and Intervention in a 
Comprehensive Literacy Classroom, which was delivered online in spring 2004 to K-3 teachers 
and principals, and K-12 special education teachers in 56 schools that received Reading First 
funding. From the state’s point of view, the opportunity allowed them to leverage federal funds 
to investigate online professional development as a strategy for helping teachers meet high- 
quality requirements. 

The course was conducted from February through May of 2004. More than 1,200 
teachers and principals participated in the professional development, and extensive data was 
collected and analyzed concerning the participants’ teaching and technology experience, content 
mastery, and attitudes. Data indicating impact on teaching practice and student achievement were 
collected and analyzed as well. Data analysis was completed by researchers at The CNA 
Corporation. 

Building on the expertise of internal staff in face-to-face professional development as 
well as multimedia and online instructional design, the course was designed to provide an online 
experience that capitalized on the best research available for online professional development. 

As White (2000) relates when writing about online teaching, “Too frequently, instructors do not 
consider the impact of course design on student learning.” This was not the intention of the 
developers. Special attention was paid to the development of learning communities and to the 
needs of adult learners. Reading-specific content and skill acquisition were emphasized over 
technical skills. Guidelines for content development included internally developed instructional 
design guidelines as well as application of Keller’s (1987, 1999) ARCS model of motivational 
design. 



Instructional Design Guidelines 

Staff at AEL have excelled in the delivery of face-to-face professional development since 
the corporation’s inception 40 years ago, and have had recent successes in the development of 
both stand-alone and online multimedia-based professional development. Based on this 
experience and the work of organizations like the National Staff Development Council (NSDC, 
2001), an internal work group developed instructional design guidelines that influenced the 
development of the online content (Table 1). These standards support corporate beliefs about 
professional development, regardless of delivery mode, and emphasize effective strategies such 
as the development of learning communities, providing sustained support over time, and 
acknowledging the needs of adult learners. 




Table 1. Instructional Design Guidelines 



Guideline 


Description 


Framework 


Learners are better able to make connections and derive personal 
meaning from content when the design includes a framework or structure 
for that content. 


Evidence 


Adult learners look for an evidence base when they approach a new 
learning experience. 


Relevance 


Relevant content and opportunities for practical applications enhance 
adult motivation and learning. 


Community 


Learning occurs best in community. 


Time 


Learning occurs over time. 


Learner background 


Effective learning designs acknowledge and accommodate the diverse 


and preference 


backgrounds, interests, and learning styles of adults. 


Activities 


An appropriate balance of content structuring (presentation), personal 
reflection, interaction with colleagues, and hands-on or applied learning 
enhances adult motivation and engagement. 


Assessments 


Formative assessment is integral to participant learning and improving 
the instruction. 



Completely self-paced instruction was not utilized during the implementation. This 
decision was reached through discussions between the state department of education and AEL, 
and based on a history of low completion rates for self-paced instruction. Building on the 
learning community metaphor, the state designated master teachers called Literacy Leaders at 
each school, who received training throughout the year on a variety of topics related to reading 
instruction and assessment. These Literacy Leaders served as facilitators for their own learning 
communities, creating what might be referred to as “group self-paced” instruction; that is, each 
community enforced its own timeline, level of participation, and acceptable levels of content 
mastery. The content included structures intended to guide the Literacy Leaders, such as a face- 
to-face and online orientation, which presented guidelines for obtaining requisite administrative 
and technical support, developing and keeping a timeline, and facilitating online learning. 



ARCS Model 

Developed by Keller (1987), the ARCS model of motivational design is widely used in 
the development of instruction in multiple fonnats (Small, 1997), including computer-based 
instruction and distance education (Keller, 1999). The ARCS acronym stands for Attention, 
Relevance, Confidence, and Satisfaction, and Keller breaks each down into three subcomponents 
(for an overview of components, see Small, 1997). 

Based on conversations with state department of education staff and AEL staff 
experience delivering professional development in a public school setting, it was postulated that 
most course participants were busy professionals, with competing demands on their time, who 
may feel burdened by participating in whole-school professional development. In addition, it was 
surmised that many of the participants had minimal technology skills and experience. It was 
thought that they needed a relevant and satisfying experience that would capture their attention 







and give them the confidence they needed to transfer the knowledge and skills from the online 
setting to the classroom. 



The application of the ARCS model played out in several facets of the instructional 
design of the course, including the structure, chunking, and sequencing of the content; the design 
and implementation of the activities; and the selection of media and alternate media formats that 
sought to satisfy a variety of platfonns, processing capacities, and connection speeds. The 
participants’ experience and current practice were critical factors as the school-based 
communities reviewed findings from research, explored their own practices, and constructed new 
knowledge within the context of their professional learning communities. The activities in each 
module often followed a three-part progression in development from: 1) reflection on initial 
practice, to 2) exploration of new concepts, to 3) synthesis of new skills and knowledge with 
current practice. These activities resulted in activities or processes to apply within the school or 
classroom. 

The content contained text, graphics, videos, and animations delivered via course 
management system (CMS) software called Course InSite, which was developed by Avatar 
Technology. Tailored for the delivery of professional development, Course InSite also supports 
individual professional development portfolios and customized administrative reporting, and 
incorporates software for threaded discussions as well as chat, e-mail, and document 
management features. All video and animations included low-bandwidth — and no-bandwidth — 
options, with a companion CD-ROM available for offline work that contained all videos, 
animations, worksheets, and handouts. Threaded discussion was a key feature for monitoring 
participation and received extra attention in the face-to-face and online orientation. Online 
discussions were also a key source of data for determining the degree of successful participation, 
as most activities required group or individual postings to the discussion list, often with 
reflection on postings by other members of the community. 



Evaluation 

The data. The tests developed for the course consist of a 68-question pretest and five 
posttests corresponding to the learning objectives for the five modules. Participants were also 
given a chance to complete an online evaluation of each training module. Web-based discussion 
boards were available to participants to help address questions and share advice and information 
on the programs. Additionally, upon completion of the professional development, Literacy 
Leaders participated in focus group interviews and a postprogram satisfaction survey. In this 
section we present findings based on analyses of these data. 



Results from pre- and posttests. A total of 1,270 classroom teachers and administrators 
took the online modules. Of these, 693 participants completed all 68 pretest and corresponding 
posttest questions. Based on this sample, the average pretest score was 64%. Average pretest 
scores across modules ranged from 51% (module four) to 75% (module five). On average, 
scores improved 2 1 percentage points overall. The percentage point gain in posttest scores 
ranged from 13% to 31% across modules, with the largest gain made on module four and the 




smallest on module five, indicating some equalization of pretest differences. Posttest average 
scores were above 80% for every module (Table 2). 



Table 2. Test Scores by Module (n=693) 





Module 1 


Module 2 


Module 3 


Module 4 


Module 5 


Total % Correct 


Correct pre 


63% 


64% 


61% 


51% 


75% 


64% 


Correct post 


83% 


83% 


89% 


82% 


88% 


85% 


Point gain 


20% 


19% 


27% 


31% 


13% 


21% 


Pet gain 


31% 


30% 


45% 


61% 


18% 


33% 


N of questions 


12 


20 


13 


9 


14 


68 



Of those reporting their race (74%), 76% were non-Hispanic Whites and 23% were 
Black. The vast majority of participants (81%) were female; 5% were male. Fourteen percent did 
not report their gender. Multivariate analysis of test score data revealed no differences among 
participants across groups (not shown). 



We found that respondents in at-risk schools scored slightly better on the pretest (not 
statistically different) and similarly on the posttest to others (Table 3). 



Table 3. Raw Test Scores for At-Risk Schools 



At-Risk School 


Pretest 


Posttest 


Gain 


Observations 


Yes 


43.95 


57.40 


13.45 


483 


No 


42.05 


58.65 


16.60 


210 



After a preliminary examination of pre- and posttest results by instructional position, we 
grouped findings into those for literacy staff (Literacy Leaders and reading resource teachers) 
and all others (classroom teachers, administrators and unknown positions). As shown in Table 4, 
pretest raw score averages were 7.5 points higher for literacy staff than for others (not 
statistically different); posttest averages for all groups were very similar. 



Table 4. Raw Test Scores for Literacy Staff and General Staff 



Literacy Staff 


Pretest 


Posttest 


Gain 


Observations 


Yes 


50.6 


56.7 


5.6 


36 


No 


43.0 


57.8 


14.9 


657 



Results from qualitative data. Accompanying each online exam is an evaluation 
component gauging participants’ opinions and satisfaction regarding module content, 
presentation, and effectiveness. These data are useful for formative assessment of the online 
Reading First program. In addition, they help to interpret results from the quantitative analyses 
presented above. Two additional, rich sources of qualitative feedback to the online program were 
derived from focus group interviews and surveys of Literacy Leaders in the summer following 
the program. 













Module evaluations 

Tabulations of the module evaluations indicate consistent, moderate-to-high satisfaction 
across the board, with nonsignificant differences between modules. Among the unstructured 
comments, the most frequent positive feedback was regarding module content (over 80%). The 
most frequent suggestions for change (38%) referred to presentation issues. One third of 
respondents indicated they would change nothing. The Web board postings were much less 
structured than the survey responses. Fully half of all postings dealt with start-up issues in 
module one. 

Literacy Leader Surveys and Focus Groups 

We found that overall satisfaction levels reported by Literacy Leaders are similar to those 
of the broader participants. Additionally, numerous and consistent comments arose regarding the 
implementation difficulties and the compressed program period. Opinions of the module content 
remained unambiguously positive. 

Data from the Literacy Leader survey also validated two of the original assumptions of 
the designers: 1) the participants felt burdened in terms of time, and 2) concerns about the 
technology were a factor for participants (Table 5). 



Table 5. Literacy Leader Survey Data Reported in Number of Responses 



To what extent did you experience the 


Not at All 


Small 


Moderate 


Large 


following implementation issues? 


Extent 


Extent 


Extent 


Lack of time for teachers 


1 


8 


18 


24 


Lack of commitment from teachers 


11 


20 


18 


3 


Lack of support from school principal 


41 


6 


4 


1 


Lack of support from district administrators 


40 


6 


5 


1 


Lack of support from state program 
administrators 


46 


4 


2 


0 


Inadequate technology training for teachers 


12 


18 


14 


8 


Unreliable technology 


13 


18 


8 


13 


Too many other school commitments 


1 


19 


12 


20 



Survey data also demonstrated that Literacy Leaders felt that while there were concerns 
about the online professional development at the initiation of the training, the reaction to the 
experience at the end was positive (Table 6). Inadequate technology training and unreliable 
technology appeared to be factors that persisted throughout the implementation. 



Table 6. Literacy Leader Survey Data Reported in Number of Responses 



To what extent do you agree 
with the following 
statements? 


Strongly 

Agree 


Somewhat 

Agree 


Neither 

Agree 

nor 

Disagree 


Disagree 


Strongly 

Disagree 


Does 

Not 

Apply 


Teachers liked the Reading 
First online PD program. 


5 


24 


7 


13 


3 


0 


Teachers prefer to receive PD 
online. 


5 


18 


11 


14 


4 


0 







This online PD program met ^ 

the needs of new teachers. 

This online PD program met 

the needs of experienced 13 

teachers. 

At the start of the program, 
teacher reaction to online PD 1 

was positive. 

By the end of the program, 

teacher reaction to the online 7 

PD was positive. 

Informal conversations to 

discuss online PD were 1 5 

important to teachers’ learning. 

Group meetings to discuss 

online PD were important to 12 

teachers’ learning. 

The online discussion boards 

were important to teachers’ 3 

learning. 

The online PD resulted in 

changes in classroom practice. 



20 


11 


2 


1 


1 


22 


10 


7 


0 


0 


12 


9 


19 


11 


0 


25 


2 


13 


4 


0 


27 


9 


1 


0 


0 


25 


12 


2 


1 


0 


16 


15 


15 


3 


0 


10 


11 


7 


1 


0 



Conclusions 

Indications thus far are of a very successful initial implementation of the online 
Assessment and Intervention in a Comprehensive Literacy Classroom. The first such indication 
is found in the comparison of pretest and posttest averages for literacy staff versus regular 
participants. As might be expected, literacy staff scored substantially higher in the pretests, but in 
the posttests the other participants caught up and achieved very similar posttest scores, 
sometimes exceeding the Literacy Leader averages. 

Related to the consistent test results is an overall satisfaction expressed regarding the 
content of the training modules. All three sources of participant feedback — online evaluations, 
threaded discussions, and focus group surveys — indicated high levels of satisfaction with the 
content of the training modules. 

Implementation problems appeared to center around two issues: 1) the introduction of the 
program late in the school year and 2) technology and software difficulties, many of which were 
addressed as the training continued. Addressing these issues in future implementations, and 
developing improved means of evaluation and feedback should be priorities of the program. In 
addition, evaluation of longer-term retention of knowledge gains and changes in instructional 
practices as a result of the program would be useful. 
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